Add run-evals orchestration skill and evals for published skills#189
Conversation
…utilizing absolute paths
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a comprehensive Skill Evaluations (Evals) Framework for AI agent skills, including documentation, universal code quality rubrics, and skill-specific evaluation configurations (evals.json) for several skills. It also adds a new run-evals skill and a Dart unit test (skills_evals_test.dart) to ensure structural consistency across all evaluation files. Feedback on the unit test suggests simplifying the type promotion logic to improve code readability.
|
Request to update readme to make it really clear the use case for these "evals" are essentially like unit tests for the skills within dart-lint-skills package and not for the agent client plugins or skills within the overall repo that this package currently lives in. Also what someone who comes across this "run evals" skills might use it for i.e. primarily used to evaluate contributions of skills related to the development of flutter framework, dart language, flutter/dart plugin/packages. |
| "id": 1, | ||
| "prompt": "Run the evals for the definition-of-done skill.", | ||
| "expected_output": "The agent orchestrates a with-skill subagent in a branched workspace, grades its git diffs and stdout, and outputs a markdown artifact.", | ||
| "expectations": [ |
There was a problem hiding this comment.
Lets change this to mean the expectations for the for what was done and the end state of the repo after. This will stay a list of requirements but will focus on changes to the repo and tracked files. Feel free to come up with a better name.
| "repo-criteria": [ | ||
| "evals/code_quality_rubric.json" | ||
| ], | ||
| "evals": [ |
There was a problem hiding this comment.
I think the evals structure might need the ability to have a starting configuration state and to specify what that state is.
Think skills that warn on dirty git starting state, evaluated skills with particular classes of errors. Different repo formats etc.
3 participants
Human description
We need a way to unit test and regression test skills "evals" are how the industry seems to do it. The evals system authored by the devtools team is focused on user journeys and end to end flows along with token efficiency and model comparisons. This problem of unit and regression tests for contributor skills and even the few published skills for a package author is on the roadmap but too far out to wait. The pattern introduced in this pr is intended for local consumption to give me some way of evaluating regression in skills and the most valuable parts are the prompts and expectations. The rest of the "infra" can and should be replaced when something better comes along.
Finally these tests do not run automatically and they may not work for your harness and model. They require being run manually and were testing using an internal version of antigravity and gemini 3.1 pro and all pass.
Agent description
Description
Authored by /contributor-pr-description feedback welcome.
This PR introduces the
run-evalsorchestration skill, adds structural consistency testing, and createsevals.jsontests for published skills in thedart_skills_lintpackage.The
run-evalsskill executes evaluations for each skill in complete isolation (using a branched workspace).Changes
.agents/skills/run-evals/: Added the orchestration skill for executing skill evaluations in isolated branch workspaces.evals/: Addedevals/README.mdandcode_quality_rubric.jsonto introduce a two-tiered evaluation architecture. This architecture separates skill-specific task assertions (defined in individualevals.jsonfiles) from universal quality standards (defined in shared rubric files). This modular approach allows us to enforce broad health and quality expectations across all agent runs without duplicating them into every skill's evaluation file.test/skills_evals_test.dart: Added a test suite enforcing that all published skills inskills/contain anevals.jsonfile. It also guarantees that everyevals.jsonfile across the repository shares the exact same JSON structure, while keeping that structure schema-agnostic and flexible. Re-architected JSON decoding and type promotion to use clean Dart 3 pattern matching switch expressions.evals.jsonCleanup: Dropped the redundant"skill_name"key from all evaluation files (as it's inferred from the parent directory) and hoisted"repo_criteria"to the top of the files to prioritize readability.Related PRs
add-dart-lint-validation-ruleskill, built on top of this PR'sevals.jsoninfrastructure.code_quality_rubric.json, ensuring the rubric correctly catches anti-patterns while permitting clean code.